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ABSTRACT 

Selection procedures for the Wellington (New Zealand) 
Polytechnic Journalism Program were studied. An exploratory analysis 
of 1983 selection data was undertaken, resulting in elimination of 
one selection test and establishment of a provisional cutoff for 1984 
applicants. In the principal study, the following four aspects were 
examined: (1) selection tests; (2) the selection interview; (3) the 
personal information asked of applicants; and (4) the role of the 
written assignment required of applicants. Selection tests included: 
a test of general mental ability for use at the tertiary level; a 
story written after interviev.ing another candidate; and a test of 
listening comprehension. The pool of about 162 successful applicants 
from 1982 to 1984 (about 54 applicants per year) and about 150 
unsuccessful applicants from 1982 to 1984 (about 50 applicants per 
year) formed the sample* Correlations were determined for the course 
grades, selection tests, and the ratings of candidates by three 
course instructors. Selection tests demonstrated satisfactory 
validity against course grades. It is suggested that the general 
ability test *MOUld be best complemented by a composite test sampling 
the skills measured by the other tests. Recommendations incorporated 
into the 1985 selection process and suggested for future use are 
discussed. Appendices provide information about students and forms 
used in the study. (SLD) 
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1.0 INTRODUCTION 



1.1 NZCER^s Involvement 

NZCER became involved in the selection of students for the Wellington 
Polytechnic Journalism Course at the request of the course tutors. Over 
the previous two years Journalism Staff (JS) had contacted the Council's 
Test Advisory Services about interpretation of a standardized test being 
used at that time (applicability of norms; stability of IQ; effects of 
varying language backgrounds on performance). During these discussions 
the validity of the test had been raised, and JS came to the view that 
their own assessments of the validity of this test should be investigated 
more rigorously. It was a short step to move from that point to a decision 
to investigate the total selection procedure in use at that time. 

The concensus of JS was that current and previous selection procedures 
had served them well, but restructuring of the journalism course and 
changes to the general nature of the pool of applicants suggested the need 
for a formal assessment of the selection procedures. If such an 
investigation led to changes in future selection procedures, these would be 
welcome. If, however, changes were not indicated, JS would continue in the 
knowledge that existing procedures had been subject to a formal evaluation. 

1.2 Prelimincan^ Investigations 

Following preliminary discussions about existing selection practices, 
an exploratory analysis of 1983 selection data was undertaken. At this 
early stage there was no attempt to relate selection tests to course marks 
or final grades; this analysis was of selection data only. By comparing 
the means and standard deviations for the 'selected' and 'rejected' groups 
of applicants, it was possible to demonstrate to JS the actual 'cut-offs' 
that had operated for that year, and to introduce the concepts of 
measurement error and the standard error of measurement. It was also 
possible to demonstrate which of the current selection tests was providing 
the most 'weight' in selection, and to show how the tests intercorrelated. 
On the basis of this exploratory work one selection test was not used with 
1984 applicants, and a provisional cut-off equal to the 1983 mean of 
'rejected' students minus one standard deviation, was instituted for the 
major selection test. 

During 1984, before the study proper commenced, the selection 
interview was also examined. ' The major feature of selection interviews 
were discussed with JS. As the interview was an important determinant of 



final selection for the course, it was apparent that steps had to be taken 
ta ensure that consistent criteria were used by all interviewers and that 
all candidates were placed on a connnon scale. These aspects of the 
interview were all the more important vjhen it is considered that there 
were a variety of interviewers, a number of locations and unspecified 
criteria for 'acceptance* or ^rejection*. 

NZCER Gtaff and JS were able to identify five broad areas to be 
covered by the selection interview, describe achievement and behaviour 
relevant to these areas, and formulate a numerical scale to accompany 
each. This draft interview schedule formed the basis of interviews for 
the 1985 course. Provision was made to redraft the interview schedule 
for the 1986 course and to embark on a validation of the revised format 
at that stage. 

1 . S Objectives of the Study 

This study was designed to investigate four aspects of the selection 
procedures • 

These were: 
(i) The selection tests, 
(ii) The selection interview, 
(iii) The nature of the personal information asked of course applicants, 
(iv) The role of the 'selection assignment'. 

(i) The Selection Tests 

The primary focus of the study centres on an examination of the 
predictive validity of the selection tests. Analyses were undertaken of 
the internal characteristics of these tests and the relationships between 
performance on these tests aiid available criteria, such as course marks 
and tutors' ratings. 

By taking sets of selection test results and course marks for 
applicants from 1982 to 1984, it was hoped to gain a more complete picture 
than could be provided by any one year- In addition, it was hoped that 
developing group profiles of 'selected' students over three years would be 
useful in establishing realistic 'cut-off scores for accepting or 
rejecting applicants. 

For the years 1982 to 1984, the following selection tests were in use. 

- A test of general mental ability eveloped for use at tertiary 
level. 

- A 'story' written as a result of interviewing another candidate, 
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A test of listening comprehension consisting of recall of the 
factual aspects of a recorded newspaper editorial. 



(ii) The Selection Interview 

The practice during the years under investigation has been to 
interview each candidate in a fairly informal and unstructured manner. 
As a result, a global judgement was made as to the 'suitability' of each 
candidate. One of the principal thrusts of this investigation centres 
on the following aspects of the interview: 

- Developing criteria. 

- Identifying behaviours indicative of the various criteria. 

- Developing a numerical rating scale to summarize interviewers' 
assessments . 

- Drafting an interview schedule incorporating the three points 
above. 

These four aspects of .the interview were identified as needing 
attention first, because of the importance of the interview in the 
selection process and the marked improvement in reliability and validity 
that would result. 

(iii) Personal Information 

Over a number of years, a comprehensive application form had been 
developed, covering a variety of biographical, educational and personal 
data. The major questions for consideration were: what importance should 
be attached to this information? what data could be shown to make a valid 
contribution to a selection decision? how could these data be integrated 
into a meaningful whole? was some form of weighting appropriate? 

(iv) Role of the 'Selection Assignment ' 

As part of the application, students are required to submit a written 
assignment based on. an interview of a person featuring in a newspaper 
article. Current practice is not to assess the assignment in any 
systematic way, but to regard candidates who do not return it as not 
having completed an applicatior: . In general, those who do not return an 
assignment are rejected at this point. The major issue to consider is 
whether the quality of the written assignment can be reliably assessed 
and used in the selection process. Is it possible to reject those whose 
writing skills are not judged to be of a certain minimum standard? 
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In summary then, the specific objectives of this study were fourfold: 

1. To investigate and report on the validity of the current selection 
tests and to make recommendations about future selection testing as 
suggested by the findings of this study. 

2. To work with JS to improve the validity and reliability of the 
selection interview. 

3. To advise J3 on the utility of the information gained from current 
application forms, with a view to suggesting how the information 
obtained could be focussed directly on the matter of selection. 

4. To consider whether the present selection assignment could or should 
contribute to an assessment of each candidate's suitability. 

1.4 Constraints of the Study 

The study was carried out with existing data from selection tests, 
course marks, tutor's ratings and interview assessments. In effect, all 
the analysis was of a post hoc nature. There was no opportunity to 
explore relationships not encompassed by the existing data. 

With the exception of the interview, the nature of the selection 
data created no particular problems. However, there were some difficulties 
validating data based on course marks and assignments. This does not 
suggest that course marks were invalid or unreliable, but reflects the 
changing priorities of JS over the three year period. For example, there 
were changes to the nature, scoring, frequency and underlying content of 
some class assessments and ratings. These changes were in response to 
issues of teaching and assessment as seen by JS. From the point of this 
research, these changes in the nature and form of the results from year to 
year meant that data, entry measures were validated against, were not 
constant. Although we could reasonably assume that all course data 
possessed satisfactory content validity, there was no way of assessing 
the impact of the changed nature of the data on the resulting validity 
•coefficients. By and large however, our assumption was that despite 
changes to the grading scales, the measurements of learning and 
achievement remained valid in relation to the objectives of the courses. 

Given the potential value of the interview in the selection 
procedure, it was s-en as important to include some data from this 
source in the validation process. Difficulties were encountered here, 
as numerical data on a 5-point scale from the interviews existed for only 
one year. Nor were there qualitative assessments of the kind that could 
be transformed to an appropriate numerical scale. Thus, no systematic 
ERJC analysis of the contribution of interview performance to predicting 



achievement in the study of journalism could be undertaken. 

This also highlights a major difference between research into 
selection, and the real difficulties of selecting candidates. Providing 
there are agreed and constant criteria, it is reasonable for experienced 
interviewers to discuss and finally decide on each applicant's suitability, 
in qualitative terms. For the purposes of research a dichotomy of 'accept' 
or 'reject' is not adequate for numerical analysis, when the sample 
already exhibits a restricted range of achievement. \^en there are no 
course data available for those who have been rejected by interview, the 
dichotomy provides no basis at all. This illustrates, however, the reality 
of basing research on data found adequate for other purposes. 

If elements of longitudinal research or in depth case studies could 
have been included, it may have been possible to tease out some of the 
biographical details contributing to course success. Indeed, the 
utilization of multiple regression analyses would also provide strong 
inferential data of this nature, but regrettably, was beyond the scope of 
this study. Hence the research team's task was restricted to considering 
what aspects of the biographical information could contribute to a 
selection decision, bearing firmly in mind that the ultimate objective of 
the selection process was to choose students who were most likely to 
complete the course to the best possible standard. 

2.0 THE JOURNALISM COURSE AND EXISTING SELECTION PROCEDURES 

2. 1 Description of Joumatism Course 

The one year, full time Journalism course at Wellington Polytechnic is 
part of the General Studies Department. The course concentrates on 
writing, news-gathering and sub-editing skills, in addition to the basics , 
of preparing material for publication. Most of the students' work is 
published in either course publications or outside publications and is 
print oriented, but there is the opportunity for some work in the 
electronic media. 

The course aims to give a sound grounding in the skills required to 
work in any area of the media, including daily newspapers, radio, public 
relations and information work. Most of the students are adult and have 
full or part degrees. (See Appendix I) 

ERJC 12 



2.2 Existing Selection Procedures 

About 400 applicants a year apply for the course, for which there 
are only approximately 60 places. Dealing adequately with such a number, 
in addition to daily tutoring tasks, imposes a considerable burden on JS, 
during the final term in particular. In 1982 and 1983 selection begaa with 
applicants attending a polytechnic in groups of about 20, where they each 
interviewed another applicant and then wrote a 'story* as if they were 
working for a small-town newspaper. This was marked by a tutor and graded 
for 'natural news-writing ability', i,e,, clarity and brevity. The 
applicants then coc Dieted the ACER B40 test. During the two hour period 
following the test, each applicant was interviewed by a tutor or media 
representative for approximately 20 minutes. In that time the interviewer 
sought to establish whether the applicant had the 'personality, general 
knowledge and inquiring mind' necessary to work in the media. Because 
such an assessment was subjective, the tutors were reluctant to rely too 
mrch on the outcome of the interview. However, interviews were felt 
necessary for 'face validity'. 

After the lunch/interview break, the applicants listened twice 
(without taking notes) to a tape on press censorship and were then asked 
to recall as much as they could. This was scored by a tutor and marks 
awarded for accurate recall of the points made. In addition, a 'plus' 
was jiven if the writing was particularly concise and clear, and a 'minus' 
if it was clumsily expressed. When all applicants had completed the 
selection procedures (news story, interview,- 4CiE7? B40 ^ Comprehension)^ the 
tutors sifted through the results, selecting the successful candidates for 
the following year's course. 

JS tended to look first at the ACER B40 and Comprehension results. 
If they were acceptable (i.e., better than about 120 and 60% respectively), 
the tutors then looked at the 'story' score and the background information 
supplied by the applicant in the interview. In looking at the background 
information JS were trying to assess motivation, but they were well aware 
how difficult this was. The interview mark was viewed with 'healthy 
skepticism' . 

Age was another factor considered and young people from the sixth or 
seventh forms had to do exceptionally well in the ACER B40 or 
Comprehension to be accepted. Generally, the tutors relied mainly on 
these two scores, regarding them as the most objective. They did this 
because results on the course over the years suggested to them that while 
good scores in these two tests did not guarantee success, it was necessary, 




with tew exceptions, for a person to have done well in the tests if they 



were to excel* However, if a person had applied more than once they were 
often accepted ahead of first-time applicants who had scored better. 
Applying more than once was regarded as indicating strong motivation. 
While some of t* ose accepted on this basis did well on the course, others 
were unimpressive. 

Tutors also exercised 'positive discrimination' in the case of 
applicants from minority ethnic groups. They did this for two main 
reasons. Firstly, the industry had indicated an interest in hiring more 
representatives of minority groups. Secondly, it was felt that the 
validity of the tests might be suspect for many minority group applicants 
who were second language learners. 

In 1984 the JS were forced to change their selection procedures 
because of the increasing number of applicants. It was decided to use a 
o'^.reening process that would reduce the number of applicants to be 
interviewed. When people inquired ^^/jout the course, they were sent a 
lengthy questionnaire to complete and were also asked to do a written 
assignment. This involved them in interviewing someone involved with the 
news, or featured in the news, and then writing a report. Those who 
completed these tasks were invited to undertake the ACER B40 and 
Comprehension tests. Those who scored reasonably well (about 117 plus for 
the ACER B40 and 60% plus for the Comprehension) and those whose 
supporting material (questionnaire and assignment) was satisfactory were 
interviewed by either a tutor or an industry representative or, in some 
cases, both. 

As a result of this procedure, lAO were interviewed from which 60 were 
selected. During the selection process more attention was paid to the 
interview results than in previous years, because JS were of the opinion 
that 1984 interviews had been conducted in a more uniform way, using an 
outline developed in consultation with NZCER staff. However, interview 
results were still treated with some caution because of the lack of 
demonstrated validity and concern about excessive subjectivity. JS were 
also, at this stage, questioning the wisdom of having media representatives 
undertake selection interviewing for the Polytechnic, without the presence 
of at least one tutor. JS thought that tutor-conducted interviews were 
likely to be more consistent because the tutors have interviewed many 
more applicants and are better able to make informed comparisons. 
Additionally, JS are able to confer often with a view to developing and 
maintaining constant standards. This, of course, will involve tutors in 
more interviewing than previously. 

One of the more significant recent developments in the selection 
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process for the course has been the increased emphasis on establishing the 
motivation of applicants. This has been achieved by ensuring t-hat serious 
applicants have had some previous experience of journalism before joining 
the course, e.g., work on a student newspaper, or radio station, or 
freelance work for a provincia]. or community newspaper. It was felt that 
people who have some prior experience of journalism are more likely to 
remain motivated when they are on the course, than others who have drifted 
into the field, in the hope that it might suit them» 

In the past all those who have completed the course satisfactorily 
have seemed to do well on the job. Very few complaints from employers 
have been received by JS about the performance of graduates. Further 
feedback from former students suggests that the skills imparted on the 
course arc highly relevant. A suirvey by the Journalists Training Board 
found that 82% of polytechnic graduates surveyed regarded their training 
as a 'good* or * excellent* preparation for the job. 

2.2 Overseas Research into Selection for Journalism Training^ 

Much of the specific research on selection for journalism originating 
from overseas must be interpreted cautiously, mainly because of 
differences between the Polytechnic Journalism course and the overseas 
courses. In the United States, for instance, journalism education is 
located largely within the College system, where it is often part of 
Communications Departments. As credits from Communications may count 
towards degrees and diplomas in a range of fields, study may reflect 
academic concerns, rather than focusing on the practice of journalism- 
Because of this academic nature of many journalism courses, selection 
criteria have tended to focus on the attributes of academic success 
rather than the attributes needed to complete a practically oriented 
course similar to the one at Wellington Polytechnic. Notwithstanding 
the difficulty of interpreting specific findings, conclusions coming 
from some American research are relevant here. 

Brown and Weaver (i979) reviewed a number of current procedures and 
concluded that the ''predictors effective in one graduate programme may 
not be effective in another*" They did note, however, that for their own 
course which "... strikes a balance between professional and academic 
training that is tipped toward scholarship", that Grade Point Average 
(GPA) was generally the best predictor of graduate grades, but the 
maximum correlation of .27 was not strong. Additionally, the Graduate 
Record Examination (ORE) Verbal was found to be a weak to moderate 



predictor of graduate GPA which did not support Scotton (1977), who had 
concluded that GRE Quantitative was generally more useful for predicting 
graduate grades than GRE Verbal. 

Brown and Weaver also indicated that more than test scores are 
needed to predict graduate success, as their best four predictors 
accounted for just 27% of the variability in graduate GPA. The unknown 
reliability of GPA probably contributed to this moderate prediction. 
Lynn (1978) found that 39% of the variability in graduate GPA was 
accounted for by undergraduate GPA plus two GRE scores, but the addition 
of scores from the Catifomian Psychological Inventory boosted the 
variance accounted for to nearly 75%. Findings of this nature led then 
to suggest that graduate admissions connnittees should take a hard 

look at applicants' personality characteristics as revealed in 
interviews, autobiographical statements, conversations with others who 
have known them well, and such tests as the Califomian Psychological 
Inventory/^ Although Lynn does not elaborate on the meaning of "... take 
a hard look at applicants' personality characteristics", this can be 
viewed as some support for the policy of including an interview in 
selection for journalism. Their suggestion for including the CPI as well 
is intriguing, but it does presuppose knowledge of relationships 
between journalism and CPI profiles. 

Scotton (1977) undertook a survey of the admission procedures f'^r 
71 college journalism programmes. He found that 46 of these programmes 
turned away at least one half of their applicants and that across the 
study there was a 20% dropout rate. After reviewing studies of GRE Verbal 
scores, GRE Quantitative^ scores^ GRE Advanced Tests f personal 
recommendation, biographical information and interviews, Scotton noted 
that this latter category had been found to be the best single predictor 
of interpersonal skills; qualities cited by several studies as being 
important to journalism. He suggested a need to re-evaluate selection 
criteria for journalism in terms of the requirements of the programme. 
If selection is to be on academic grounds only, the prediction of 
'success' will be minimal, if the prograimne incorporates skills other 
than those sampled by academic measures- In a word, selection procedures 
must be valid. 

Payne, Wells and Clarke (1971) have- noted that the reliability of 
various predictors of success change from programme to prograimne. This 
suggests that the definition of success may differ from programme to 
programme, or that success is being defined in a way that does not 
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relate logically to admission criteria. This may be more true of 
postgraduate journalism programmes where the emphasis shifts from 
academic criteria at entry, to professional development as the course 
progresses. Whatever the reason, the outcome is a lack of validity in 
the selection tests, which is manifested in terms of varying reliability, 

Fedler and O'Keefe (1977) describe screening procedures used to 
select students to undertake practical assignments on a daily paper during 
their course work. They report that their screening procedures improved 
the calibre of students going forward and ensured that students without 
basic reporting competencies, did not become a burden to newspaper staff. 
Prior to implementing the screening procedure, the internship programme 
with the local paper had been restricted to students who earned a 'B' or 
better in basic reporting and editing courses, but these assessments did 
not prove to be adequate by themselves. The screening procedure began 
with an application from all students wishing to participate. The 
application consisted of academic transcripts, examples of academic work 
and published stories. Phase two involved both typing and style tests. 

The significance of the selection procedures described by Fedler and 
O^Keefe is probably to be found in the manner in which they sampled daily 
skills of journalism. In effect, they instituted assessments based on 
'work samples'. Traditionally, 'work samples' have proved to be a most 
valid approach to selection, but the most difficult problems associated 
with them revolve around the actual assessment of the sample, and the 
time taken to obtain the 'product'. An additional important consideration 
for the Fedler and O'Keefe procedure Is that it differed from a typical 
situation of primary selection for course entry. Their students had 
partly completed a journalism course, so they had the opportunity to master 
a core of journalism skills utilized in the screening procedure. The 
screening then could measure developed skills, and not have to rely on 
identifying candidates who might develop skills given appropriate 
instruction. 

What then are some implications of these studies for the Polytechnic 
Journalism Course? The importance of ensuring that selection measures 
sample skills that underpin course performance is of primary importance. 
As a corollary to this, goes the need to match the selection to the course 
and submit selection procedures to an empirical validation* The other 
major implication for this study is the support for adequate assessments 
of applicants' personality characteristics, by interview or other means. 
^ This is a conclusion that JS had reached themselves and fully supports the 
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emphasis placed on- improving the validity and reliability of the interview. 
The specific finding reporting the contribution to selection made by a 
personality test, is food for thought. It further supports the role of 
personality attributes in Journalism training and indicates an alternative 
measure that could be investigated. 

3.0 METHODOLOGY 

3.1 The Sample 

The total pool of completed applications for 1982 to L984 formed the 
basis for the study. Applicants were identified as 'Successful', that is, 
were selected for and completed the journalism course, or 'Unsuccessful', 
that is, were rejected, or did not complete the course. For each year, 
complete records for all Successful applicants (N » approximately 54 each 
year) were gathered. A random sample of approximately 50 Unsuccessful 
applicants were selected for each year. 

Over the three years there was reasonable uniformity of selection 
measures administered, allowing comparisons to be made of both selected 
and rejected groups on these measures. Some variation did exist for 
assignments and end of course assessments, meaning that comparisons 
between selection results and course results for the three groups of 
selected students are not identical. Although these 'criterion 
variables' differ, they do reflect differences in the structure of the 
course from year to year, so they may be regarded as equally valid 
measures of achievement. 

3.2 Selection Tests and Course Measures 

A brief description of selection tests and course measures follows: 

(i) ACER Advanced Test B40 (second edition) 

A broadly based, predominantly verbal measure of general mental 
ability. Designed for use at tertiary level and having maximum 
discrimination around IQ 120. Provides percentile ranks, IQ's and 
stanines. Reliability and validity known. 

(ii) Comprehension 

A test of recall of facts as presented in a recorded newspaper 
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editorial. Each candidate is required to summarize the exerpt after a 
second listening. Note taking is not permitted. Scoring is based on the 
number of facts recalled, with an additional two-point rating indicating 
'satisfactory' or 'unsatisfactory' writing style • Marking undertaken by 
one tutor. Hitherto no information on reliability or validity, 

(iii) ABBA 

Applicants are paired off. Each member of the pair interviews the 
other person, then writes a short report of the interview. The report 
of the interview is marked on a variable scale by one tutor for 'writing 
style'. No information on reliability or validity available. 

(iv) Interview 

A free ranging selection interview of about 20 minutes duration, 
covering no fixed content and without standard criteria. Interview may 
be conducted by JS, media representatives or a combination. No provision 
for s um marizing or reporting on candidates in a unifonri manner. No 
information on reliability or validity. 

These selection measures remained constant for 1982-1984 candidates. 
Modifications were introduced for candidates for the 1985 course* 

2 Course Marks 

The typical pattern of course marks and grades was for each student 
to be assessed on a core of six subject marks, plus two of four additional 
optional marks. A final summative assessment was also issuec Some 
assessments, e.g., typing and shorthand gave scores representing words per 
minute, while the others were generally graded on a five point (A-E) scale 

For 1982, letter grades (converted to a 10 point numerical scale) for 
ten individual subjects were available. The topics included: reporting, 
subbing, principles, attitude, shorthand and typing; with special options 
in public relations, sub-editing, broadcasting and feature writing- For 
1983, marks for up to twelve topics were available: newsgathering, 
newswriting, subbing, feature writing, investigative, business reporting, 
shorthand and typing; vith special options in public relations, sub-editin 
broadcasting and feature writing- A summative mark of all course topics 
was computed by standardizing each score, summing across subjects and then 
averaging- This procedure was followed to ensure that each topic 
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contributed equally to the composite score. For 1984, the course mark 
available was a global one with credit pass/pass/fail grades awarded for 
performance on the course as a whole. This grade was converted to a three 

point numerical scale. 

In addition to course marks for 1983 and 1984, the three course 
instructors independently rated each student for" likely success as a 
journalist. Ratings were provided on a ten-point scale. The three 
ratings for each student were then pooled to provide an overall tutor- 
rating. These ratings were gathered some months after the end of the 
1984 course. 

3.4 Analysis of Data 

All necessary transformations and coding of data were carried out by 
the research team. Data tapes were prepared at the Polytechnic Computer 
Centre for i.ubsequent analysis at NZCER, using the Statistical Analysis 
System (SAS). 

♦ 

3. 5 Development of Revised Interview Schedule 

The nature of the selection interview used for selecting students for 
1982-1984, has already been described in broad detail. During late 1984 
some modifications were introducea prior to the selection round for 1985 
applicants. 

The dissatisfaction JS expressed with the format and procedures of 
the selection interview up to and including the 1984 Course, proved to be 
the starting point for a close examination of procedures, and the 
development of a standard interview schedule. Before specific work 
started on the selection interview for journalism, JS undertook some 
reading on selection interviewing, and engaged in discussions with the 
authors about the concepts of validity and reliability ai\d, in particular, 
their applicability to selection interviewing. From the outset, the 
development of the interview schedule and the formulation of interviewing 
procedures were shared between JS and the authors. JS provided their 
expert knowledge of journalism, the journalism course, and the 
characteristics, interests, attitudes and backgrounds of students, 
. suggested by their collective experience to be important for successful 
completion of the course. The authors helped JS identify and articulate 
all the factors that would contribute to the validity of the final 
interview. On the basis of this information a standard interview 
0_ schedule incorporating appropriate numerical scales, was drafted. 
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For the restructured interview, six areas of behaviour were 
identified as being valid indicators of likely success on the course. 
These were Writing and Publication Experience; Knowledge of Journalism; 
Awareness of Media Issues; Intellectual Curiosity /Investigat.^ve Skills; 
and Awareness of Issues of Social Justice in New Zealand Society. A 
summarizing category of "Overall Suitability was also included. Each of 
these categories was accompanied by a five-point numerical scale with 
verbal descriptors. (See Appendix II) 

In addition to the five broad areas to be focused on during the 
interview, a series of 'anchor' questions was developed for each section 
of the interview. The anchor questions were designed to make up 
approximately 75% of the interview, thus leaving interviewers scope to 
explore aspects appropriate to the background and perceived talents of 
individual candidates. Given that these were 'selection' interviews as 
defined by Bolton (1983), it was hoped that the provision of anchor 
questions would ensure that all candidates would receive broadly 
equivalent interviews. 

Having regard for the well known rating problems of errors of 
leniency, errors of central tendency and the halo effect (Guilford, 1954), 
it was decided to construct a numerical /graphic scale to provide a 
measure of each behavioural category identified. Given the likely 
multidimensionality of the set of traits to be rated, numerical summation 
was not to be used to determine a final rating. Validation of the 
interview schedule was not possible prior to the beginning of the 1985 
selection round, so the development of the interview to this point was 
regarded as part of a continuing research investigation. Results of 
interviews for those selected for the 1986 Course are to be followed up 
during 1986. 



4 . 0 RESULTS 

4. 1 Suaaessful and Unsuaaessful Appliaants 

The test performance of Successful and Unsuccessful applicants for 
1982-1984 is summarized in Table 4.1. 

The groups of Successful applicants have performed better than their 
corresponding Unsuccessful counterparts. . The former obtain significantly 
higher mean scores than the -latter, on the ACER B4G , and the ABBA tests. 
This pattern is consistent from 1982 to 1984. 



TABLE 4.1 Comparison of Successful and Unsuccessful Applicants 
on the Selection Tests (1982-1984) 







Successful Applicants 


Unsuccessful Applicants 




Year 


Test 


iMean 


b 


Mln 




iMean 


c "n 

b . U 


Mln 


nax 


t 


1982 


ACER B40 


127 . 7 




11/ 


1 J/ 


1 lo . Z 


/ . J 


iUU 


1 97 


9.75* 




ABBA 


7 . U 


1 . U 






0 . 0 


1 1 


J 


o 

o 


1.90 




Comp 


12.7 


2.9 


0 


1 n 
19 


7 . 5 


J. J 


1 

X 


ij 


8.52** 




Interview 


21 .4 


2.9 


1 c 

15 


25 


2U. U 


4 . 5 


lU 


Zj 


1.89 


1983 


ACER B40 


128.5 


5.3 


116 


141 


118.9 


9.3 


93 


133 


8 . 24** 




ABBA 


5.8 


1.1 


3 


8 


5.2 


1.2 


2 


8 


2.61* 




Comp 


7.1 


1.3 


4 


9 


5.7 


1.5 


1 


8 


5 . 00** 


1984 


ACER B40 


124. Ir 


7.6 


100 


136 


120.4 


6.6 


97 


132 


2.59* 




ABBA 


6.6 


0.8 


5 


8 


5.9 


1.0 


2 


8 


3.89** 




Comp 


7.1 


2.2 


2 


11 


5.1 


2.0 


1 


9 


4.76** 



* Statistically significant at p < .05 
Statistically significant at p < .01 



Further, the groups of Successful applicants tend to be more uniform 
(i.e., homogeneous) in terms of ability as represented by the substantially 
smaller standard deviations. This is especially so on the ACER B40 for 
1982 and 1983. The operation of ^positive discrimination* in 1984 
resulted in a somewhat more heterogeneous group of successful students. 

To examine further the relative performance of the Successful and 
Unsuccessful groups on the selection tests, the distribution of scores 
on each measure was examined. The overlap between the minimum scores of 
the Successful applicants and the maximum scores of Unsuccessful 
applicants is reported in Table 4.2. 

Despite the statistically significant differences between the mean 
scores of the two groups on most measures, a substantial proportion of 
applicants come within this area of overlap. Between 55% (1982) and 98% 
(1984) of our samples of rejected applicants have scored at ov above the 
minimum for the accepted applicants. Again the most pronounced overlap 
occurred in 1984 when a policy of 'positive discrimination' was adopted, 
which meant that a small number of second language applicants who didn't 
meet all test criteria were included in the course. The overlap was 
almost complete for the ABBA and Comprehension tests with approximately 



95% of the rejected "jroup gaining scores at or above the minimm of the 
accepted group. 



TABLE 4.2 Number and Peroentage of Unsuaaessful AppHaants Who Scored 
At or Above Suaaessful AppHaants on Three Seleation Tests 



Selection Test 


1982 
N(%) 


1 n o 1 

1983 
N(%) 


-1984 
N(%) 


ACER B40 

at or above maximum of successful group 
at or above mean of successful group 
at or above minimum of successful group 


29(55%) 


5(10%) 
34(71%) 


16(33%) 
47(98%) 


ABBA 

at or above maximum of successful group 
at or above mean of successful group 
at or above minimum of successful group 


30(58%) 
50(96%) 


1(2%) 
19(40%) 
47(98%) 


1(2%) 
11(23%) 
■ 45(94%) 


COMPREHENSION 

at or above maximum of successful group 
at or above mean of successful group 
at or above ninimum of successful group 


3(6%) 
40(75%) 


30(63%) 
45(94%) 


12(25%) 
47(98%) 



4.2 Selection TestSy Course Marks and Tutor Ratings 

(i) The Selection Tests 

Table 4,3 summarizes the correlation coefficients between the 
selection tests for the total sample of applicants (above the diagonal) 
and for the group of successful applicants (below the diagonal) . 

Because of the more homogeneous nature of the latter group, the 
correlations below the diagonal are lower overall than for the sample 
as a whole. However, for both groups, the relationship between the 
ACER B40 and the Comprehension test is consistently the strongest and 
statistically significant. This indicates that there is more in common 
between these two tests than between any other pair of tests in the 
selection battery* 
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TABLE 4.3 Correlations Between the Selection Tests 



Test 


1 


1982 

2 


3 


4 


1 


1983 
2 


3 


1 


1984 
2 


3 


1. ACER B40 




,31* 


.79* 


.18 




.39* 


.55* 




.19 


.55* 


2. ABBA 


.13 




.30* 


.00 


-.01 




.27* 


.20 




.23* 


3. Comp 


.39 


.18 




.14 


.27 


.08\ 




.53* 


.27^ 




4. Interview 


-.08 


.05 


.05 

















(total sample above the diagonal; the selected group below the diagonal) 
* Statistically significant at p < ,05 



An important requirement for test instr\iments is reliability, that is, 
the test must produce consistent results from one occasion to another. The 
ACER B40 has a reported reliability coefficient of .85 (KR-20) . Estimates 
of the reliability of the other two tests was also obtained by applying the 
KR-20 formula to test statistics. Va^ Comprehension test had a modest 
degree of reliability (approximately 0.5), while no statistical measure of 
the reliability of the ABBA test was possible. It appeared to have nil 
reliability - a serious defect. 

(ii) Course Marks 

In order to examine the relationships between the selection tests and 
course marks, a series of correlational analyses was carried out. The 
correlations between the selection tests, course marks and ratings are 
reported in Table 4.4. As outlined previously, it should be noted that 
the course marks differ from year to year. 

The relationship between the selection tests and course marks for 
1982 and 1983 vary from moderate to weak (.55 - .07). This may be a 
result of the nature of the course marks and the procedure for determining 
them. However, since the practice of awarding grades in individual 
subjects no longer operated in 1984, there is more value in examining the 
figures for this year. The course marks and instructors' ratings a]l 
correlated at a statistically significant level with the selection tests 
for 1984. The ACER B40 shoxi/ed the strongest correlations with course marks 
and ratings (.56), while the Conrprehension and ABEA were slightly lower 
with .47 and .53 respectively. The correlations between ABBA and tutor 
O ratings are worthy of mention too. These are either highest or equal 




TABLE 4.4 Correlations Between Selection Tests and Course Grades 



Selection Test 


Course^ 


Course^ 


Rating 

1 Q 0 '5 


Course^ 
1984 


Rating 
1984 


ACER B40 


.55* 


.26 


.41* 


.56* 


.47* 


ABBA 


.28* 


.21 


.41* 


.53* 


.53* 


Comprehension 


.27 


.07 


.23 


.47* 


.47* 


Interview 


.05 











Correlations corrected for coarse grouping and restriction of range 
^ Course mark = average standardized mark over individual subjects 
^ Course mark =• credit/pass/fail mark awarded as overall grade 
Rating =» composite value of ten-point scale from three tutors 
* Statistically significant at p ^ .05 or greater 

highest for 1983 and 1984, and have been recorded despite the almost total 
unreliability of 455^4. 

It should be noted that correlations can be influenced by a number of 
factors, such as the coarseness of the grading scale, the reliability of 
the tests and the distribution of abilities of the members of the group. 
While the correlations were corrected for coarseness of the scale, the 
true relationship between selection tests and course marks may still be 
underestimated because of the relative unreliability of the test measures 
and the homogeneous nature of the groups. 

(iii) Raters and Ratings 

Some interesting findings of secondary interest in this study, relate 
to the level of agreement that exists between the three course instructors. 
Whilst each instructor teaches independently, all three must collectively 
provide an end-of-course assessment incorporating the. assessments made of 
students over many topic areas. All instructors teach aspects of each 
topic and frequently teach the same material to different subgroups of 
students. Hence, it is relatively important that there be a high level of 
agreement. 

It can be seen from Table 4.5 that the three instuctors provided 
ratings which were of similar level (mean) and spread (standard deviation) 
and, in addition, were very much in agreement (median correlation of .80 
O between individual raters). The pooled ratings show a median value of .95, 
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indicating a high level of relationship between each rater and his/her two 
colleagues. 



TABLE 4-5 Correlations BetDeen Ratings of Three Tutors (19d2'-1984) 



Year 


Rater 


Mean 


S.D 


Correlation with 
Rater 2 Rater 3 


Pooled 
Ratines 


1983 


Rater 1 


2.9 


.80 


.85 


.73 


.94 




Rater 2 


2.9 


.81 




.81 


.98 




Rater 3 


2.8 


.91 






.96 


1984 


Rater 1 


•2.6 


1.00 


.69 


.80 


.93 




Rater 2 


2.5 


1.20 




.82 


.97 




Rater 3 


2.8 


.81 






.95 



Correlations corrected for coarse grouping 



TABLE 4.6 Correlations Betueen Tutors' Ratings and Course Marks 
(1983-1984) 



1983 1984 



Rater 1 


.79 


.97 


Rater 2 


.80 


.76 


Rater 3 


.87 


.91 


Pooled Rating 


.94 


.99 



Correlation corrected for coarse grouping 



Table 4.6 contains data on the relationship between tutors' ratings 
and course marks for two years. 

As might be anticipated, the relationship between instructors' 
ratings and the final course marks awarded were also extremely high, with 
median values between individuals of 0.91 for 1984 and 0.80 for 1983. The 
latter is slightly lower, probably due to the one year longer period 
between the two assessments. The pooled rating for all three tutors is 
even higher where the racings almost parallel their course marks, 
indicating that the marks and the ratings share a common basis. 
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4.2 Interview Data 

The only objective data available regarding the validity of the 
interview were presented in Tables 4.1 and 4.3. These indicated that for 
1982 there was no measurable difference between scores awarded for the 
interview for successful and unsuccessful applicants and that the 
relationship between the inteirvipv and the three other selection measures 
was practically nil. It is highly likely that these data reflect the 
lack of reliability of the interview at that time. The interview was 
subsequently restructured as described in an earlier section. 

5.0 DISCUSSION 

5.1 Successful and Unsuccessful Applicants 

It is zlear from the analysis of t -st results for 1982-1984, that the 
selection measures in use differentiate effectively between the Successful 
and Unsuccessful groups of candidates. With the exception of one test in 
one year, all measures showed statistically significant differences, when 
means for the two groups were compared. This is not surprising when it is 
reiterated that in the first instance at least, selection is based on the 
results of the tests. This does emp. le that the objective test data 
were applied consistently enough to re uit in two distinct groups with 
measurably different characteristics. In no case has the mean for the 
Unsuccessful group exceeded the mean of the Successful group. In other 
words at the broadest level, the selection process has been consistent. 

In all but two cases, the standard deviation for the selected group 
is less than the standard deviation for the unselected group, illustrating 
that students accepted for the course were more homogeneous, with respect 
to the skills measured, than those who were not. With the exception of 
1984, when a 'positive discrimination' policy for some second language 
applicants was followed, there tends to be a greater difference between 
the minimum ACER B40 scores for each group, than the maximum scores, 
indicating that the greater variability i^ largely due to the presence of 
candidates in the Unsuccessful group, who scored about two standard 
deviations below the mean of that group. This suggests there are 
applicants who either do not appreciate the highly selective or 
competitive nature of the course, or do not have a realistic appreciation 
of their own abilities or background. 

Despite the substantial differences between the mean performances of 



the Successful and Unsuccessful groups, of more importance from a selection 
point of view, is the overlap between the minimum scores of those selected 
and the maximum scores of those rejected. Clearly there are a number of 
applicants who fall into this category, despite the significant 
differences between the mean scores of these two groups. Between 55% 
(1982) and 98% (1984) of our samples of rejected candidates have equalled 
or surpassed the lowest score from the group of the accepted candidates. 
The proportions vary over tests and years, but they are most often a 
function of the lowest score on any single test by an accepted candidate. 
Additionally, between 6% and 63% of rejected candidates scored beyond the 
means of the accepted students for any one test. 

At first glance the proportion of rejected candidates placed above 
either the minimum or mean score for accepted candidates on any one test, 
seems large. A major confounding factor is th;^t in any one year up to 
about ten candidates who are offered places reject the offer in favour of 
sone alternative. On administrative grounds, these candidates are finally 
assigned to the rejected group, although in terms of entry tests, they 
have met the criteria for acceptance. This group will have inflated the 
test data we have collected on the Unsuccessful group, and thereby depress 
the already substantial differences we have recorded. 

When a multi-stage selection procedure is being used, a number of 
rejected individuals may exceed the minimum score of the selected group 
for that variable, but the problem arises when they fall below the minimum 
value for some other variable. For example, we can identify quite clearly 
candidates who scored above the minimum ACER B40 score for the Successful 
group, but who also scored below the minimum of this group for 
Comprehension. Rejection was on the basis of this Comprehension score, 
despite the ACER B40 result. Also, we were able to identify other 
candidates who scored more than a standard deviation above the minimum 
Comprehension score for the accepted group. Their rejection was on the 
basis of the ACER B40 score. 

The total number of 'false negatives' may appear to be accentuated 
when multiple measures are used in selection, each having a different 
cut-off point. One alternative is to accept all candidates who satisfy 
any single criterion. Another is to accept only those candidates who 
surpass all the individual selection criteria. A third approach is to 
adopt a summed and weighted score approach, as developed by Angoff (I98A). 
This has an additional merit of allowing superior performance on one 
measure, to compensate for a more average performance on another. 
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However, if there are multiple measures, 'false negative?' will occur 
in some form or other, when a fixed quota of applicants is to be accepted. 
The aim must be to minimize the number of candidates in this category. 
The paradox for selection is that by increasing the number and range of 
quality measurements, the validity and reliability of the overall 
selection process should be enhanced, but at the same time/ i:he proportion 
of partial 'false negatives', i.e., candidates above the cut-off for some 
measures only, will rise also. 

5.2 Selection Measures and Course Marks 

The statistically significant relationships found between selection 
tests are dominated by the ACER B40 and the Comprehension test, 
indicating that of the measures being used, these two share most in 
common. Some significant relationships were found between the 
Comprehension test and ABBAf two tests incorporating a substantial writing 
component. ABBA and ACEE B40 shared the least common variance. Interview 
data, available for 1982 only, was nonsignificant. 

As would be expected, the relationships between test measures were 
more pronounced when they were calculated for the Successful and 
Unsuccessful groups combined. For the selected group, the relationship 
between ACER B40 and Comprehension was significant for two of the three 
years. No other statistically significant relationships were recorded 
within the selected group. Given that Comprehension proved to be of 
moderate reliability, the significant correlations between ACER B40 and 
Comprehension are noteworthy, as lack of reliability on the part of either 
measure imposes a ceiling on the resultant correlation. 

The problem of restriction of range is commonly encountered in 
studies incorporating analysis of data from a selected group. Because the 
total range of abilities present in the population of applicants is 
truncated by selection, statistics based on regression cannot demonstrate 
the full relationship that existed prior to selection. The result of this 
restriction of range is illustrated by the generally lower levels of 
correlations between tests for the selected group, as compared with the 
total group. 

With the exception of 1984 results, the values of the correlations 
have about halved, although Comprehension and ACER 840 continue to show 
the strongest relationship. The much lesser reduction of correlations for 
1984 does not mean that the relationship between Comprehension and 
ACER B40 was intrinsically stronger that year, but rather that the 1984 




policy of 'positive discrimination' resulted in a selected group, with a 
wider range of measured abilities than was the case for other years. 

It has been shown quite clearly that the students making up the 
Journalism Course have achieved the highest selection marks. Their course 
marks too, tend to bunch towards the upper end of the marking scale. 
Despite the skew of selection marks and course grades, there were still a 
number of significant relationships between selection tests, course marks 
and ratings. Results for 1984 again show the strongest relationships, 
more than likely reflecting the greater heterogeniety of that group. 
ACER B40 and ABBA show the greatest number of significant relationships 
between course marks and ratings, with Comprehension next. The strong 
showing of the ABBA test, despite its complete lack of reliability, 
suggests the importance of having a measure of writing retained in the 
selection battery. The slightly less strong showing of the Comprehension 
test, suggests that a combination of the writing thrust of ABBA and the 
cognitive emphasis of the Comprehension test, could provide the basis for 
a valid selection measure. 

On the whole, the relationships between selection tests and course 
marks for 1982 and 1983 are only moderate, despite corrections for 
restriction of range. It is likely that the nature of the course mark 
derived for this study, is largely responsible. For 1982 and 1982, the 
course mark was derived by averaging standardized marks across ten to 
twelve individual subjects. For 1984, the course mark was an overall 
grade awarded by JS. The combination of the overall grade, and the 
greater range of the group's measured abilities, are the most likely 
reasons for the substantial increase in the relationships for this latter 
year. Notwithstanding all considerations, the statistical relationships 
are strong within the context of a selection study, and undoubtedly 
represent the situation for 1984. 

In addition to course marks, global ratings by three tutors also 
served as criteria for judging the validity of the selection test results. 
Overall, correlations between the selection tests and tutor ratings are a 
little higher, and a little more consistent than correlations between 
these tests and course marks. This leads to tl\e question of which measure 
is the more valid indication of course outcomes. On logical grounds, more 
weight should be given to the ratings, as they were undertaken from overt 
criteria, were arrived at independently by three raters, show high levels 
of agreement, and are a measure of broad course outcomes. The course 
marks, on the other hand, were not comparable from year to year, were on 




differing scales, were of unknown reliability and were measures of specific 
aspects of the course, rather than general outcomes. 

The relationship demonstrated between ABBA and the ratings are worthy 
of some additional comment. ABBA correlated more highly with ratings for 

1983 and 1984 than did the other two tests, despite its total lack of 
measurable reliability. Is the rating of * likely success* really another 
measure of writing skill, or is writing skill the dominant quality leading 
to success in Journalism? Would the ratings of 'likely success' hold up 
against on-the--Job assessments? Have we uncovered an enduring measure, 
that predicts course and job performance, or are these assessments 
primarily a reflection of the experience and skill of JJ? Most of the 
above probably contribute a little, but these particular results further 
underline the need for a valid and reliable measure of writing to be 
included in any selection for this course. 

The selection tests have demonstrated satisfactory levels of validity, 
against the criteria of course ratings. Taking this relationship and the 
intercorrelations between the tests into account, it is likely that the 
ACER 840 could be complemented best by a composite test sampling the 
skills measured by ABBA and Comprehension. 

5. 3 Raters cmd Ratings 

There was a high level of agreement between raters, indicating that 
they were operating from standard criteria. With respect to ratings and 
course marks, average levels of agreement were very high. The ratings for 

1984 were a little higher than for 1983, but this probably was a function 
of recency, as both sets were generated for the purpose of this study. 

The high level of agreement between course marks and pooled rankings 
should be a source of considerable satisfaction to JS, as they show a 
level of validity seldom obtained in educational measurement. The value 
of pooled ratings is also well illustrated, as the values of these 
correlations are higher than the individual correlations. 

Taking the total package of selection tests, course marks and course 
ratings, it is apparent that we have a set of valid measurements 
functioning in a consistent fashion. The validity of the selection tests 
has been demonstrated and the adequacy of the criteria against which they 
have been judged has, we believe, been established* Our data suggest that 
it is feasible to reduce the number of selection tests and, at the same 
time, bring about some improvement in validity and reliability. 



5.4 CHterion Cut-Off Scdres 

A number of factors need to be considered before recommending 
criterion cut-off scores on the ACER B40 and Comprehension. 

First, it is assumed that the overall calibre of applicants is 
relatively constant from year to year, with a mean and standard deviation 
on the ACER B40 of about 122 and 8 respectively, and on the Comprr.hension 
test of about 60% and 20% respectively. 

Second, the maximum number of applicants that JS may reasonably 
expect to be able to interview is up to approximately 130. Allowances 
also need to be made for the operation of •positive discrimination*, in 
which members of ethnic minorities who do not meet the selection test 
criteria are accepted. About five or six such places are reserved, as 
long as JS consider each applicant can successfully complete the course. 
Another category of applicants which needs to be taken into consideration 
includes those who meet the selection test criteria, have a successful 
interview and are offered a position in the course, but subsequently 
withdraw. Approximately ten applicants each year fall into this category. 

Of the selection tests, the ACER B40 has been shown to have clearly 
greate; predictive validity than the Comprehension or ABBA tests. However, 
the inclusion of an alternative test which incorporates components of the 
latter two, is very important for reasons of face validity, even though 
its addition is likely to improve the prediction of success in the course 
only moderately. 

The primary purpose of the selection tests under investigation, is to 
reduce by valid means, the number of applicants (from an initial pool of 
250 to 400) to approximately 130, whp are then interviewed. On the basis 
of the interview, 55-60 applicants are offered a place on the course, and 
10-15 will go onto a provisional list to replace those who withdraw- 
Those eligible for preferential entry are considered separately. 

Cut-off procedures on the selection test to identify the top 130-140 
applicants will vary with the size of the original applicant pool. For 
instance, when the number of applicants is 400, more stringent cut-offs 
will be necessary than when the number of applicants is 300. Table 6.1 
shows the percentages to be accepted and the corresponding ACER B40 
cut-off which would identify the top 120 for interviewing, for applicant 
pools from 200 to 450. In addition to these applicants, those who score 
at or beyond 75% on the proposed Journalism'^ test, would provide a total 
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* This figure needs to be verified as the Journalism test is a proposed 
new measure combining the Comprehension and ABBA tests employed in the 
previous three years. 32 



of approximately 130 for interviewing* 

6.0 CONCLUS::ONS AND RECOMMENDATIONS 

6.1 What Tests Should be Used in Selecting Journalism Students? 

A number of factors must be considered in making judgements about the 
most appropriate testing strategy to use in selecting for the journalism 
course. Face validity of the procedure is an important consideration, as 
the process must be seen to be appropriate to journalism. While the 
ACER B40 is clearly the best single indicator of success on the journalism 
course, use of this alone to select applicants may well be viewed as 
inappropriate, because it assesses predominantly academic abilities, and 
ignores others considered 'journalistic'. 

By including in the selection procedure a journalistic-type exercise 
such as the ABBA^ or some other measure of reporting, it is likely that 
both face validity and predictive validity will be improved. The analyses 
presented earlier indicate that the ABBA or similar writing task, would 
add more to the prediction of success on the course, than would 
Comprehension. However,, as stated ea*:lier, the reliability of ABBA was 
nil, while that of Comprehension was moderate. Hence, the ABBA test in 
its present foim could not be considered to provide any usable information 
to aid selection. As described earlier, the ABBA test involves the 
reporting of an interview and cc isequently may represent a somewhat 
different task in terms of difficulty for each student. That is, some 
interviewees may be more forthcoming and therefore easier to interview 
than others, hence the write-up may be easier too. This would seem to 
create inconsistencies in the requirements of the exercise and hence 
adversely affect the reliability of the test scores, as well as 
influencing the ultimate validity. 

With some modification, however, a Journalism test could incorporate 
elements or Comprehension and ABBA. A revised'.test would need to 
represent the same task for all applicants, be administered in a 
standardized manner and be scored objectively. Such a test might involve 
listening to an address/report, during which note taking is permitted. 
The test could assess the comprehension of the address, and require 
applicants to 'write up' the address, as though it was a report for 
publication. The report could be assessed for both content and style on 
scales that would provide useful informrtion for selection purposes. 
(See Appendix III) g 



Deciding on the nature of tests to be included in the selection 
process is one matter, but of an equally pressing concern, is the 
development of valid criterion cut-off scores. 

Some broad issues relating to these scores were discussed earlier, 
but before firm conclusions can be drawn, the following constraints and 
assumptions must be specified: 

- the number of applicants JS are available to interview will be 
approximately 130. The final selection of 60-65 students, 
including reserves, will be made from this pool. 

- the overall calibre of applicants is constant from year to year, 
i.e., ACER B40 x - 122, s.d « 8; Comprehension (or its new 
equivalent) x = 60%, s.d » 20%. 

- allowances are to be made for 'positive discrimination' , by 
which members of ethnic minorities who JS consider will succeed, 
are found places on the course. 

- all information from or about applicants, should contribute to 
the selection process. 

The actual value of cut-off scores on the ACER B40 must finally be a 
function of the size of the applicant pool. For example, from Table 6.1, 
for an applicant pool of 300 (i.e., those who had completed the initial 
assignment and taken the selection tests), a cut-off score of 124 would 
apply. An additional 5-10% may be included by adding applicants gaining 
a score of 75% or better on Comprehension or its new equivalent. Thus 
making the number available for interviewing approximately 130. 

TABLE 6.1 Number of Applicants Related to ACER B40 Cut-Off Scores 



Number of Applicants 
Taking ACER 340 


Percentages Accepted 


Cut-off on 

ACER B40 


up to 200 


60 


120 


201-225 


60-53 


121 


226-250 


53-48 


122 


251-275 


48-44 


123 


276-300 


44-40 


124 


301-350 


40-34 


125 


351-400 


34-30 


126 


401-450 


30-26 


127 
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We acknowledge that, at this stage, we are being somewhat tentative 
about the role of Comprehension in selection, as we intend to reconnnend 
that it be modified to include elements of ABBA and. be renamed Journalism. 
Our reasonable assumption is that the validity coefficient for this new 
measure will not be less than those obtained for Comprehension* 

6. 2 What is the Role of the Interview ? 

Given that we have virtually no data on previous interviews, we have 
been obliged to start from square one. The modified interview format and 
anchor questions developed as part of this study, appear to have adequate 
consistency in the hands of trained interviewers. Given the pooled 
expertise and experience of JS, it is likely that the interview will 
prove to be a valid indicator of course success, assuming that the factors 
measured by the interview are i levant to the course. This must remain to 
be put to the test. 

The initial justification will be that JS are convinced of the 
predictive worth of the interview. There is support for this view in the 
overseas studies cited earlier. Validation and refinement of the 
interview can be demonstrated in objective terms. If this validity 
ultimately falls below acceptable leX^els, decisions abgut the fate of the 
interview must then be made. It is the intention of the project team to 
follow up the interview in 1986, but we should make our final position 
clear. Unless the reliability and validity of the interview can be 
demonstrated, it has no place in the selection of young people for career 
training. Given these circumstances, drawing names from a hat would be 
equally effective. 

g. J The Use of Information from the Application t'orm 

No data on the use of information in the application form was 
obtained, hence the coverage of this objective of the study has been 
minimal. However, use of information obtained from the application form 
and design of the form was discussed. JS did indicate that a considerable 
amount of time was spent assessing the infonration presented. The 
conclusion reached was that developments in the application form over a 
number of years had resulted in a document that obtained much information 
that could not be used effectively in selection, because it had net been 
established whether much of this information was a valid indicator of 
course success. Hence the application form was redesigned with the aim 
^ of making all information relevant to selection for the course. 
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Although this guiding principle was accepted, the point was made by 
JS that seemingly irrelevant questions on an application form could 
provide information to be explored further during an interview. The 
redesigned application form is iucluded in Appendix IV, 

6.4 The Selection Assignment 

The selection assignment was a relatively new procedure, first being 
introduced in 1984 for reasons that could not be clearly substantiated. 
Our view was that it could contribute to the selection decision if it was 
assessed in a reliable fashion. There was also the possibility that the 
selection assignment could provide a measure of writing and thus 
substitute for one of the foxnnal selection measures. 

Administrative problems have beset the selection assignment. In 1985 
for example, one group of applicants were not required to complete this. 
JS have acknowledged that the nature of the assignment may make it more 
difficult for some applicants than for others. This is because the 
assignment involves the applicant in making contact with the writer or 
person featuring in a published article, and interviewing one or the other. 
Applicants who know someone in the media may find it easier to make the 
initial contact, as may those living in urban areas, because of the more 
ready access they have to media personnel. If it is agreed that these two 
factors are significant, they could be nullified by changing the nature of 
the assignment. 

If the assignment is to be a requirement of the application process, it 
should have a high degree of ^face validity', be required of all applicants, 
and, in logistic terms at least, be of about similar complexity for all. 
Additionally, it should have a potential for contributing to the selection 
process. If information coming from the assignment is then to be included 
in selection decisions, it must be assessed in a reliable fashion and its 
validity should be tested empirically. However, if the assignment is not 
used to provide objective inforraation relevant to selection, in our view, 
it should be discontinued. 



6, 5 Recommendations 

As a result of the study described in this Report, the following 
recommendations regarding the selection for the Wellington Polytechnic 
Journalism Course are made, 

(1) Tests should not be administered nor interviews conducted until after 
Q the closing date for all applications. This date should be widely 



promulgated and constant from year to year. 

(2) After this date, the following multi-stage selection process should 
begin. 

(i) Initial screening be undertaken on the basis of the application, 
provided that JS can identify criteria that may be empirically 
validated. Information from this study suggests that factors 
such as age, educational qualifications, evidence of published 
writing, prior involvement with journalism, could be considered. 
Criteria used at this point must be overt, objective and 
available if requested. In the event that it is not possible to 
develop appropriate criteria, all candidates completing an 
application should proceed to the testing stage, 
(ii) ACER B40 to be continued with. Candidates below a prescribed 

cut-off, as indicated in Table 6.1, to be rejected at that point, 
(iii) Comprehension test to be reformulated as a Journalism test. 

Candidates below a prescribed cut-off, as indicated on page 27, 
to be rejected. (See Recommendation 4) 
NOTE: Candidates who have not been rejected at this stage should be 
regarded as exhibiting the requisite verbal/educational 
attributes to complete the course, 
(iv) Only candidates not rejected at this point to be interviewed. 
All interviews to be conducted and assessed using modified 
format, scale and anchor questions, 
(v) Decisions on selection to be made on the basis of interviews. 

In the event that candidates cannot be adequately discriminated 
at this point, final weight should be given to ACER B40 results. 
(The above sequence is illustrated schematically on the following page) 

(3) Test ABBA to be discontinued in its present form. 

(4) The proposed Journalism test, combining elements of ABBA and 
Comprehension be developed. This test to be marked for recall/ 
understanding of content and quality of writring. 

(5) The selection assignment, as presently constituted, be discontinued, 
if there is a need to gather additional biographical data as a 
background to the interview, tnis could be included in the 
application form. 

(6) If the procedure of 'positive discrimination' is to be continued with, 
the selection tests and interview be used to place these candidates 

in an order of merit. Entrance to the Course should not then be 
automatic for these candidates, but on the basis that JS consider that 
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each individual will successfully complete the course. 

(7) If it is necessary to grade students* course performance for 
awarding merit passes, this be done on the basis of the combined 
ratings of all tutors, 

(8) The heavy additional burden that selection procedures place on 

JS should be brought to the attention of the Principal, Wellington 
Polytechnic. As selection is now an established aspect of the 
course, an appropriate time allowance for these duties could be 
considered for JS. 

(9) Validation of the revised interview schedule and the JoumaHsm 
test (if adopted) be undertaken during 1986. 

Proposed Selection of Journalism Students: Procedure 



Application Forms 
Received 



±. 

Assignment Done? 
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APPENDIX I: STUDENTS COMPLETING JOURNALISM COURSES, NUMBERS TWENTY 
YEARS AND OVER, AND EDUCATIONAL QUALIFICATIONS 



1982 



1983 



20 
30 
50 



19 
27 
46 



31 



30 



10 
11 
3 

16 



14 
11 
3 

18 



31 

16 
2 
1 



34 
10 
1 
1 



STUDENTS COMPLETING COURSE 

Male 

Female 

Total 



NUMBER TWENTY YEARS AND 
OVER AT END OF COURSE 



HIGHEST TERTIARY 
QUALIFICATION AT ENTRY 

Full Degree 
Part Degree 
Other 
Nil 



HIGHEST SECONDARY 
QUALIFICATION AT ENTRY 



HSC or Bursary 
University Entrance 
Sixth Form Certificate 
Other 
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APPENDIX II: INTERVIEW SUMMARY 



Polytechnic Journalism Course 
Interview Summary 



Applicant . , 
Interviewer 



Date 



Writing and Publiaation Experienae 

Extremely 
Experienced 



Very 
Experienced 



1 



Experienced 
3 



Reasonably 
Experienced 



Not 
Experienced 



Knowledge of Journalism 

Extremely Very Reasonably 

Knowledgeable Knowledgeable Knowledgeable Knowledgeable Knowledgeable 
1 ■ ■ 2 3 4 5 



Not 



Awareness of Media Issues 

Very 
Awari^. 

2 



E>:trem9ly 
Aware 

1 



Aware 
3 



Reasonably 
Aware 



Awareness of Issues of Social Justice in New Zealand Soaiety 



E::tremely 
Aware 

1 



Very 
Aware 



Aware 
3 



Reasonably 
Aware 



Intellectual Curiosity / Investigative Skills 



Extremely 
Curious 

1 



Very 
Curious 



Curious 
3 



Reasonably 
Curious 



Overall Suitability for Course 



Extremely 
Suitable 

1 



Very 
Suitable 



Suitable 
3 



Reasonably 
Suitable 



Not 
Aware 



Not 
Aware 



Not 
Curious 



Not 
Suitable 
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NOTE: 



Accompanying 'anchor' questions have not been included in Appendix, 
but readers engaged in selection should approach the Senior Tutor 
in Journalism, Wellington Polytechnic, if they wish to discuss the 
relationship of the questions to the interview schedule. 
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APPENDIX III: SCALE FOR RATING QUALITY OF WRITING 



1 « No subbing needed. Could be published as is. 

2 » Minor subbing needed before suitable for publication. 

3 » Moderate subbing needed before suitable for publication. 

4 » Major subbing needed before suitable for publication. 

5 - Needs total rewrite. Cannot be salvaged by subbing. 



No Subbing Minor Subbing Moderate Subbing Major Subbing Needs Rewrite 
Needed Needed Needed Needed Not Subbing 
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APPENDIX IV: APPLICATION FORM 



Polytechnic Journalism Course 
1987 Application Form 

Please read the questions carefully before completing. 

Family Name * 

First Names 

Name known by Date of Birth Age 

Contact Address • 



• Telephone 

Ne.\r. of Kin: Name Telephone 



APPLICATION FOR COURSE 

Have you applied for this course before? Yes/No 

If YES, please give year(s) 

Have you applied, or do you intend to apply, for 

admission to another journalism course in 1987? Yes /No 



If YES, specify whic>^ course (s) 

Specify any non- journalism course (s) you have applied for in 1987. 



PHOTOGRAPH 

Please attach a recent photograph 
of yourself (passport type) 



With which race do you identify? 



ACADEMIC RECORD 

Copies of all qualifications attempted, e.g., »cby..l certificate, must 
accompany this application, or be forwarded as aoon as they are rf>c.eived. 

SECONDARY EDUCATION 



School (o) Attend^'' 



Achievement Summary: 





5 


Form 
6 


7. 


SC 
Mark 


Sixth Form 
Certificate 
Grade 


UE 
Mark 


Bursary/ 
Scholarship 

































































































































Examinations attempted in 1986 



TERTIARY EDUCATION 



Please list any course(s) attempted since leaving school, e.g., university, 
nursing, secretarial. 
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Institution 


Subject or Course 
(please list subjects, 
if degree) 


Year 


Grades 











































WORK RECORD 

Please indicate all fulltime and regular part-time employment. 



Position 


Employer 


Date 
From To 


Reason Left 



















































Have you previously worked in a branch of the media? Yes /No 

If YES, was it: [ ] fulltime or [ ] part-time? 

Name of Employer • 

Were you an: [ ] employee, [ ] freelancer, [ ] school student 

[ . ] other please specify 



Have you had any work broadcast or published? Yes/No 

If YES, please list below the stories you have had broadcast or published, 
and attach transcipts or copies. 



Story 


Publication 


Date 





































































JOUKKALISM AS A CAREER 

When did you decide you wished to become a journalist? 



Who and what influenced you to make this decision? 



What do you think you have to offer journalism? 



List you community activies and interests. 



Which of the above activities do you intend to continue in 1986? 



PROGRAMME AT WELLINGTON POLYTECHNIC 

How did you hear about the course at Wellington Polytechnic? 

Did you know any students before you applied? Yes /No 

Do you wish to work in a particular branch 

of the media after graduation? Yes /No 

If YES, state which one 

Why have you chosen this area? ' 
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GENERAL 

List the characteristics which you frel best describe yourself. 



List your favourite writers. 



What type of publication do you most enjoy reading? 



What TV programmes do you watch regularly? 



What daily newspapers do you read regularly? 



What weeklies do you read regularly? 



What do you listen to on the radio? 



What current issues have you been following in the media? 



What are the things you like or dislike about the media? 



ERIC 



47 



Most reporting jobs require a driver's licence. 

Do you have one? Yes /No 

Do you have any medical condition which may impair 

your performance as a student or a journalist? Yes/No 



SELECTION ASSIGNMENT 

1. Select a st.ry from your local newspaper. Interview either the 
reporter responsible or a person named in the story. Write a 
story based on your interview (maximum 500 words). Use A4 
paper. Type or write on one side only. Write the time, date 
and duration of the interview at the top of your story. 

2. Staple a copy of the original story to your story. 



SELECTION 

You will be required to attend selection tests and possibly an interview 
during the second half of the year. These are held in some centres 
outside Wellington. A small fee is charged for tutor travel and 
accommodation. 

Wellington tests and interviews are arranged days and weeks apart, so you 
will have to make two trips to Wellington Polytechnic. If you choose a 
venue outside Wellington, you should be prepared to stay overnight - tests 
will be run one day and interviews the next. 

Please circle the venue you prefer: 

Wellington (July) 

Wellington (August) 

Wellington (September) 

Wellin&::jn (October^ 

Wellington (November) 



Auckland (November) 
Hamilton (November) 
Chris tchurcb (November) 
Dunedin (November) 
Palmerston North (November) 



Other (no guarantees, so please specify 

an alternate) 

Alternate 
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